1. Field of the Invention
The present invention relates generally to instructional and testing technology, and specifically to methods and apparatus for more accurately, flexibly, and completely analyzing the knowledge of one or more individuals and enhancing the future performance thereof.
2. Description of Related Technology
The societal benefits of accurately evaluating the knowledge or skill level of one or more individuals are manifold and well known. Several prior approaches to evaluating an individual's knowledge or skill exist. These approaches generally include: (i) multiple choice questions; (ii) true/false questions; (iii) short answer or “fill-in-the-blank” questions; (iv) essay or “free-form” answer questions; (v) real world or practical problem solving; or (vi) personal interview. Of these approaches, multiple choice, true-false, and short-answer questions comprise the great majority of components in modern testing instruments, due to inter alia their ease of administration, relative lack of subjectivity, and comparatively short length.
However, multiple choice, true-false, and short-answer questions are often not highly probative of deficiencies in an individual's knowledge or skill level. Specifically, prior art multiple choice, etc. exams often yield little more information than whether or not an individual has enough knowledge, or guessed adequately, so as to have met the minimum requirements associated with the exam (e.g., provided a sufficient percentage of correct responses in relation to the total number of questions asked). Other prior art exam techniques monitor the time taken to respond to a question, utilizing this information as some indicia of performance.
Furthermore, prior art techniques often are not adaptive to particular individuals or groups; i.e., the techniques do not themselves “learn” about an individual's (or group's) deficiencies and tailor their subsequent operation so as to enhance the testing and learning experience for the individual/group.
The following are generally representative of various prior art testing and evaluation approaches.
U.S. Pat. No. 6,364,667 entitled “Techniques for mastering a body of knowledge by writing questions about the body of knowledge” to Heinberg, et al (“Heinberg”) has the individual write questions for a test that is given to a group of test takers. The results of the test are then analyzed to determine how well the person's questions discriminated between those who did well on the test and those who did poorly. A cycle of study, writing questions, analyzing the results, and determining mastery is repeated until the desired degree of mastery is reached. The techniques of Heinberg, however, are directed to test validation (i.e., how well the test/question discriminates between those who have varying degrees of mastery of the subject matter), and not on the evaluation of the individual's skill level or attributes/weaknesses.
U.S. Pat. No. 6,361,322 entitled “System and method for improving a user's performance on reading tests” to Linden Henry (“Henry”) discloses a method of teaching reading comprehension which includes presenting a reading passage to a user, asking the user a question relating to the reading passage, and receiving an answer from the user. The method indicates to the user that the answer is an incorrect type of answer to the question if the question tests the user's ability to identify information from the passage and the answer uses information that is not from the passage. Also, the method indicates to the user that the answer is an incorrect type of answer to the question if the question tests the user's ability to infer a conclusion from the passage and the answer uses specific information from the passage. The invention of Henry, however, is geared only to reading comprehension evaluation, and utilizes a specific answer structure related to the material in the reading passage provided. No mechanisms or techniques for accurately evaluating an individual's knowledge level of particular subject matter and adaptively probing the individual for areas of weakness are disclosed.
U.S. Pat. No. 6,210,171 entitled “Method and apparatus for multiple choice testing system with immediate feedback for correctness of response” to Epstein, et al (“Epstein”) discloses an answer form for use in answering multiple-choice questions which is configured to provide immediate feedback to an examinee as to whether a correct response has been made for each question, and if not, permits the examinee to proceed to determine the correct response. The answer form includes a substrate such as a paper surface upon which is printed a varying number of rows and columns of answer alternatives. The rows correspond to the questions to be answered, and the columns correspond to the number of answer alternatives for each question. Each alternative is covered by an opaque coating that can be rubbed off to reveal whether a particular alternative conceals an indication that the alternative is correct or not. A blank space beneath the opaque coating indicates an incorrect alternative. The invention of Epstein, however, provides no adaptive or interactive evaluation of the student's knowledge level, but rather merely provides immediate feedback of the correctness or error of the student's answers. Furthermore, Epstein's invention requires manual intervention by the student, thereby disrupting the examination process. Furthermore, positive or negative reinforcement of the student's performance is necessarily provided when the student reviews the correctness of his/her answers, thereby potential introducing additional variables into the evaluation process, and potentially distorting the results (i.e., if the student knows they are performing poorly, they may become stressed and perform even more poorly due to the stress, thereby distorting any determination of their actual skill/knowledge level).
U.S. Pat. No. 5,779,486 entitled “Methods and apparatus to assess and enhance a student's understanding in a subject” to Ho, et al (“Ho”) discloses an educational method and system that generates individually-tailored tests based on (and whose difficulties are geared towards) the student's level of understanding in the subject. One goal of Ho is to allow the student not only to use the tests to prepare for an examination, but also to use the tests to learn the subject. The invention disclosed in Ho includes a score generator that is coupled to a recommendation generator. The recommendation generator includes an inference engine and/or a pre-requisite analyzer. Both the pre-requisite analyzer and the inference engine in the recommendation generator can generate recommendations based on the student's test results. The recommendation generator is coupled to a report generator and a question generator. The report generator accesses a report format. Based on the recommendations and the report format, the report generator generates a report, which can provide assessment of the student's understanding in different areas of the subject, and which can provide action items to improve on the student's understanding in the subject. The question generator, based on the recommendations, generates a number of questions. The student can take this new set of questions to further enhance their understanding in the subject. The invention of Ho, however, does not facilitate accurate and adaptive analysis of individual question subject matter and types. Ho discloses only comparatively simplistic grading and weighting routines and inference generation based on so-called “expert” analysis.
Based on the foregoing, there is needed improved methodologies and apparatus for encoding exam questions or problems, and testing/evaluating the skill or knowledge level of subjects. Such improved methodology and apparatus would ideally allow for probative, real-time, non-intrusive, and detailed evaluation and analysis of an individual's skill or knowledge level in one or more topics, as well as adaptive structuring of subsequent examination material based on the aforementioned evaluation and analysis. Such improved methodology and apparatus would also ideally be implemented in a medium which facilitates easy (and repeated) use by anywhere from one to a multitude of individuals disposed at one or more geographic locations.